User Testing + User Research 


At the Internet Archive 



User Testing Collection on Archive.org 

Roadmap, documentation, UX Reports 
https://archive.org/details/usertestinqandresearchcollection 

UserTesting.com 

UN: *******@ arc hive.orq 
PW: 

SurveyMonkey.com 

*************** 

PW: 

A List Apart 

Articles ‘for people who make websites’ 

http://alistapart.com/ 

Nielsen Norman Group 

Amazing reports on user research 
https://www.nnqroup.com/reports/ 


Cheat Sheet 



Gives you a fuller picture of how your Ul is utilized by users 

Learn how a variety of users interact with a Ul and place a focus on ‘dark 
parts’ of the Ul that might not seem confusing to experts (or the people who 
built/designed the interface). 

Don’t fight or guess, test 

When in doubt (or in a heated debate) try testing with users. 

A vital part of QA and iteration 

Test and evaluate changes with usability testing! Plan new changes findings! 
Test multiple options against each other! 


What is UX and 
how can it help? 



Types of User 
Testing + User 
Research 


Usability Testing 

10-30 minute moderated or unmoderated task based tests of an interface or 
feature. Can be performed in-person or remotely. 

Discount Usability Testing Methods 

Heuristic evaluations (holistic overview based on a list of heuristics) + 
Cognitive Walkthroughs (task-based evaluations) are performed by UX 
evaluators not users. 

Surveys and survey-based prototypes 

Short surveys are a great way to get quantitative feedback from users. 

User Interviews / User Research 

Focused on use cases and user needs. These interviews may include 
watching a user interact with an interface unguided. 

Long Term Studies 

Diary studies and shadowing users. In Diary Studies users are asked to 
perform a task or set of tasks and then fill out a survey or write a diary entry 
daily over a period of time. Shadowing is generally used when the 
environment around the user is important to understand. 



Usability Testing 

An introduction to tips, guidelines and tools 



Generally, just 5 

For the vast majority of studies you’ll find about the same amount of usability 
problems with 5 users as you would with 30 users. 

You may want to test with 5 users in each user type (new user, novice user, 
expert user) or with 5 users per interface if testing multiple Uls. 

15 users for surveys or qualitative feedback 

If you’re looking for statistics over insights and want to reach a ‘statistically 
significant number’ you’ll want to have at least 15 users. I generally only do 
this for surveys, card-sorting or 5 minute user tests. 

‘Bad’ user testing beats no user testing 

The value of talking to just one user is greater than putting off user testing in 
hopes of recruiting a large set of users. Start small and test often. 


How many users? 



200 

150 

100 

50 

0 



Number of Test Users, Nielsen Norman Group Consulting Projects 


Usability Testing 
Tools 


UserTesting.com 

A service that recruits and administers unmoderated remote, task-based 
user testing. You will need to write up a task-based test, have a working 
prototype or live-web accessible interface. 

Pros: Quick turnaround on tests, no need to recruit users yourself, videos 
can be annotated or turned into clips or clip reels for sharing. Users are from 
across the US, Canada and England. 

Cons: Not free so please check-in with Alexis before using credits on the 
UserTesting.com account! Additionally the quality of testers can be varied. 


Talky.io + Quicktime 

Use Talky.io to face/screen chat with your tester and record your screen with 
Quicktime. 

Pros: Free, records face + screen. 

Cons: Files can be gigantic, you’ll need to recruit users yourself and 
moderate the tests. 



Recruiting Users 
and Soliciting 
Feedback 


Soliciting Feedback from a Diverse Sample of Users 

Our current feedback system places the burden of sharing feedback with the 
Archive on the user (via email, social media or forums). This makes it hard to 
learn what is working well (rather just what isn’t working) as well as places an 
advantage with vocal users or user groups. Don’t just let vocal user groups 
define feature planning or you will build a website for one (loud) user type. 

Testing with New Users 

Gaining perspectives from new users can help to shine a light on pain points 
that current users have gotten used to, and potentially may have created 
workarounds for. Fresh eyes can really help us to understand the mental 
model that we are presenting to users on their first visit and can lead to 
invaluable insights. 

Recruit Widely (this might cost money + time) 

It’s great to recruit for testing at lunch or events, but it’s also great to recruit 
via Craigslist or your social groups. The less a person knows about the 
Internet Archive the better for first time user tests. 





User Research 

An overview of methods and introduction to surveys 



Diary Studies 

In Diary Studies users are asked to perform a task or set of tasks and then fill 
out a survey or write a diary entry daily over a period of time. 

User Interviews 

User Interviews, unlike Usability Tests do not have to center around a set of 
tasks, but can instead be a conversation or formal set of questions to ask a 
user. User interviews can be great ways to spend time with users and gather 
feedback in a non-guided way about your interface or tool. 

User Surveys for Quantitative Feedback 

Short surveys are a great way to get feedback from users in a way that can 
be translated easily to a quantitative metric. 

Focus Groups for Qualitative Feedback 

Focus groups with groups of 5-10 users can be a way to gather qualitative 
feedback and highlights group dynamics. Some users might also be more 
comfortable speaking among peers rather than 1:1. 


User Research 
Methods 



When to perform 
User Research 
over User Testing? 


Creating a New Feature or Interface from Scratch 

User research is a great way to understand how users interact with live 
tools/interfaces in the same realm that you are building in. 

If Time or Multiple Interactions is a Key Element in Your Research 

For tools that require multiple uses or use over time, a diary study can help 
you get a better understanding of how users interact with the interface you 
are testing. Perhaps you want to know why users haven’t adopted regular 
use of a tool? 

You’re Unsure of What to Work on Next or Want General Feedback 

User interviews can be great ways to spend time with users and gather 
feedback in a non-guided way about your interface or tool. 

Alternatively, You Need a Lot of Lightweight Feedback 

Lightweight surveys (5-10 minutes) can be a great way to get a lot of 
quantitative feedback on a proposed change or question that you and your 
team have. The click logic in survey software also makes it easy to perform 
discount A/B testing . 





Sharing Your Research 

Creating reports geared towards interest 



People Get Distracted: Write an Executive Summary (or TL;DR) 

I always start my reports with a one pager that has the MOST important 
information I found from testing. 

Sum Up What The Study Entailed 

Share the number of users, where you recruited them from and if possible 
link to the full recordings / survey responses. 

Group Suggestions / Findings by Type or Feature 

Some people might only be interested in a specific part of the whole Ul, so I 
like to group feedback/suggestions by type and feature. It’s okay to be 
redundant because your report is probably going to be read in chunks. 

Don’t Forget to Share What Worked 

User testing doesn’t have to be all doom and gloom, it’s also a great way to 
celebrate successes. 


Sharing your 
research with your 
team 



